Support reading skims from Parquet files, in addition to OMX - #1101

@amsamimi I cannot formally make you a GitHub "reviewer" on this unless you accept the invitation to join the ActivitySim org, but since this addresses the issue you opened, you should definitely review it anyway 😉

Thanks for reviewing this @amsamimi.

At our recent engineering meeting, we also discussed that functional correctness is a necessary but not sufficient criteria for acceptance of new features; performance impacts are also relevant. While your smoke test is valuable, it would also be important to know if the new code is performing adequately. For this PR it may be possible to check that simply from the logs of the smoke test -- the main effect is at skim loading, which happens similarly for a full model run or this tiny piece of one. Can you look and see what the time needed to load parquet skims is, and compare against the equivalent OMX time on your same underlying data?

@jpn-- @janzill @joecastiglione

I ran a full-skim test with a sample size of 10,000, using a single processor and Sharrow, under two scenarios: OMX and PARQUET. The memory/runtime profiles are attached, along with the corresponding log files.

Key Observations:

OMX Version:

OMX_activitysim.log
OMX_mem.csv

PARQUET Version:

PARQUET_activitysim.log
PARQUET_mem.csv

A few notes:

• The tally of "Writing Skims to Shared Memory" at 21 sec vs 7 min is misleading; parquet is faster but not quite that much faster. I set about this PR with the singleminded instruction of "faster". This is achieved by reading parquet in parallel streams as fast as possible, but this analysis highlights that we are not piping those data streams directly to the final shared memory destination, we're just loading into whatever memory allocation makes it possible to do with maximum speed. So while it is technically correct that 21 seconds is the time it takes to get the skim data into shared memory, that is a RAM to RAM copy, and does not include read from disk or decompression. A more apples-to-apples comparison would include all the read from disk and decompress time, which is effectively the whole initialize_landuse step. Under that measure, we are looking at 2.5 minutes vs 7 minutes. Still a positive bump but not quite as large. And that 4.5 minutes difference is effectively the only difference between these two runs with respect to runtime.

• It seems likely that with some careful planning about memory use, I should be able to find a solution were we can both have and eat the cake: faster loads, but directly into the shared memory space with no giant temporary array allocation. Doing this the right way may also deliver improved speed to OMX inputs as well. I already started working towards this in sharrow#87 but I am not done testing and evaluating whether that indeed works as intended.

• I recommend we do not hold this PR until my theorized improvements can be proved out; users can decide if they have the RAM to pay the price to get this acceleration today, and then one day soon via another PR we might make it even better. I defer to @ActivitySim/product to review the current performance metrics and trade offs outlined above and decide.

Thanks @jpn-- . I've updated the summary table to avoid any misunderstanding.

Another consideration is that initialize_landuse is not the memory bottleneck. While it has a local peak in memory usage, this does not have any practical consequences, at least for our use case.